LAYER 1: EXECUTIVE CONTEXT (Cody's Notes)

LAYER 2: CORE PHILOSOPHY (The Narrative Summary)

At the heart of kqr's thesis in Software Never Fails is a fundamental shift in how we conceptualize system incidents: software never actually fails. While this assertion may initially sound counterintuitive, it rests on a strict definition of failure. A component failure is an event—a transition over time from a state of functioning as designed to a state of dysfunction due to wear, physical degradation, or external stress. Software, however, is pure logic and design. It lacks physical form and does not degrade with time or use. When software produces an undesirable or catastrophic outcome, it is not because it transitioned from a working state to a broken one; rather, it executed the exact instructions compiled into it. Thus, the bug is a constant fact of the design, present from the moment of deployment. Shifting our perspective from "software failure" (an event) to "design error" (a constant) is critical because it changes our response from blaming the software to auditing the design.

Traditional safety engineering analyzes risk by decomposing systems into individual components and predicting how each component might fail. This works well for physical parts like valves or fan blades, but completely breaks down when applied to software. In complex systems, accidents often occur when every individual component functions perfectly as designed, yet their interactions produce unwanted "emergent features." For instance, in the historic accidents analyzed (such as the Ariane 5 explosion or the Toyota throttle control issue), the software modules did exactly what they were instructed to do, but under specific operational environments, their interaction with hardware or human operators produced hazardous states. Traditional safety metrics and component-level decomposition miss these issues because interactions disappear when components are analyzed in isolation. Reliability cannot be achieved simply by compiling a list of component failure rates and assuming they are independent.

For technology leaders, this realization directly redefines the approach to Site Reliability and Production Stability (Pillar 5). A common response to reliability concerns is to introduce redundancy—such as replica databases, redundant network links, or standby servers. However, while redundancy is highly effective at mitigating independent component failures, it is ineffective against design errors. If a database is designed with an emergent flaw that causes it to consume disk space with logs under certain commands, a backup server will replicate the same data corruption or execute the same erroneous command, failing in exactly the same way. Furthermore, adding replica databases introduces complex synchronization and replication protocols, which themselves create new interaction vectors (such as split-brain scenarios, race conditions, or write-confirmation timeouts) that can take down the system without any component failing. Reliable systems cannot be achieved through simple redundancy; they require system-theoretic hazard analysis (such as STPA) that models and controls the interactions between components.

This system-theoretic perspective must also extend to the human operators—the "wetware." In traditional safety analysis, human errors are often written off as random, unpredictable failures. However, human decisions are not random; they are shaped by training, feedback loops, and the design of the operating environment. Just as software executes its compiled instructions, humans act based on the information, pressures, and constraints of their environment. Treating human errors as symptoms of a poorly designed system, rather than random components failing, forces us to redesign the system to improve the collaboration between humans and machines. A truly robust and stable production environment must be designed to tolerate human mistakes, providing operators with the visibility, feedback, and safety margins needed to successfully manage complex, emergent situations.

Generated for the Product Growth Program.

LAYER 3: FULL REFERENCE (Raw Article Content)

Software Never Fails

One of the most confusing ideas in system-theoretic accident analysis is that software never fails. Taking this perspective helps us design more robust software.

The brief reason we can say software never fails is that software always does precisely what we told it to do. The outcome may be undesirable, and it may be unexpected, but the software itself never deviates from the instructions it was compiled into.

If software contains a bug, it’s not something it did to itself. All bugs we suffer because we put them in there. For every 10–100 lines of code you write, you add a bug.[^1]

The normal response to this is to shrug it off. We’re going to do something controversial: we’re going to take the perspective that the software hasn’t failed – it did what we designed it to do.

Sure, hardware can fail – there can be tin whiskers and brownouts and cosmic ray bit flips in memory – but that’s not the software failing. The software is pure design. A design can contain errors, but it cannot fail. The ink on the paper can degrade with exposure to environment, but this is again the hardware representation failing, not the design itself.

This perspective helps us because it forces us to adopt methods of analysis that reveal potential accidents triggered by nothing at all failing – an extremely common problem in software systems.

To understand this, we first need to know what a failure is.

Failures are events, design errors are constant

A failure is the event that occurs when a component first does what we designed it to, then something happens, and then the component no longer does what it was designed to do. The failure is the event that caused the component to go from working to failed.

• A fan blade wearing out and breaking off is a failure, because the fan started out holding together and it was indeed designed to stay in one piece, but now it is in two pieces, and no longer performs its job.[^2]
• A capacitor that starts bulging is a failure, because the capacitor was designed to store a particular level of charge at its operating voltage, but now is not able to do that.
• A microchip developing a tin whisker that leads to a short circuit is a failure, because the chip worked before the whisker made contact, and then doesn’t do the right thing afterwards.
• Memory corruption due to cosmic rays is a failure, because the chip was designed to hold a bit as long as it is powered, but then it holds the wrong bit.[^3]

The consequences of failure can be reduced with redundancy. This is why we have redundant network links and power supplies in our data centres, why error-correcting RAM stores data redundantly, and why we have a passive standby database.[^4]

In contrast to failures, an error in a design is a constant fact of that design, rather than an event. Even if the design flaw is revealed in some catastrophic loss, the loss did not cause the design to go from good to flawed – it was flawed all along.

Software, being pure design-stuff, does not wear out over time, nor change itself randomly into a non-working configuration. Whenever software does not work, that error was always in the design from the first minute of deployment, even if it took some time for the error to be revealed.

What people mean by software failure

To ensure I’m not cherry-picking software accidents to support my argument, I asked an LLM to list ten high-profile software accidents in recent history.[^5] In each of these cases, you will find people claiming that “the software failed”. I will argue that in all cases, the software performed exactly as designed.

In some cases, explicitly requested features contributed to the loss. In other cases, the software was designed with emergent features nobody wanted, and those features contributed to the loss. Either way, it is always a feature designed into the software that contributes to the loss.

1. Therac-25 (1985–1987): “Medical radiation machine software bug killed 3 patients, injured 3 others with massive overdoses.”

The race condition in the software didn’t suddenly appear as a random failure. It was consistently reproducible. The Therac-25 had been designed with an emergent feature: the fast-typing operator could enable a high-power, focused electron beam mode. This is an undesirable feature when there is a patient on the other side, but it was not a random failure – it was an emergent functionality of the design. It was built into the Therac-25 and could reliably be used intentionally, and unfortunately also accidentally.

1. Ariane 5 rocket (1996): “Integer overflow error caused $370 million rocket to explode 37 seconds after launch.”

The nozzle control software was designed to interpret any signal from the intertial reference systems as a navigation signal, so when the IRS module started outputting a diagnostic bit pattern, the nozzles were deflected according to it and the rocket attitude became unfavourable. The nozzle control software did not experience a random failure; it did exactly what it was programmed to do. The rocket was designed with an emergent feature: when the IRS is shut down, lose attitude control. This is not a desirable feature, but it was there in the design all along.

1. Mars Climate Orbiter (1999): “Unit conversion error caused $327 million spacecraft to burn up.”

Against documented suggestions, one module got implemented to use pounds as the unit of force, whereas the rest of the spacecraft used newtons. There was no conversion step, so impulse numbers expressed in pound-seconds got interpreted as newton-seconds. This is not a random failure, nor the result of wear. There was no event where the module went from working to broken during spacecraft operation. It was broken from the start, it just wasn’t revealed until it needed to be inserted into Mars orbit.

1. Toyota unintended acceleration (2009–2010): “Software bugs caused sudden acceleration, recalled 9 million vehicles.”

The watchdog that resets the electronic throttle control CPU if something on it misbehaves was designed such that it would be unable to detect if the task that manages throttle angle calculation crashes. When that task has crashed, the car believes the throttle is held at some angle it is not. There is speculation there may have been a hardware failure involved in the crash of the task, but the software did exactly what it was designed to do: if the task crashes, assume fixed throttle control angle. Terrible feature? Sure. Design error? Yuppers. But not a software failure.

1. Knight Capital (2012): “Trading algorithm glitch lost $440 million in 45 minutes, nearly bankrupted the company.”

A feature flag was toggled on by an operator who did not know one of the servers gated old code behind that feature flag. The old code executed the trades it was designed to. Nothing was broken here – everything worked exactly as designed.[^6]

1. Heartbleed (2014): “OpenSSL bug exposed passwords and sensitive data across 17 % of secure web servers.”

The OpenSSL library was designed with an emergent feature where cleverly crafted heartbeat requests would allow reading server memory. Nobody wanted this feature, to be clear, but it was there all along. It was a design error, not a failure. There was no event where a running instance of OpenSSL went from working to broken. The memory reading capability was built into it from the first minute of deployment.

1. Equifax Breach (2017): “Unpatched Apache Struts vulnerability exposed data of 147 million people.”

The software that scanned for suspicious network activity had been designed to let traffic through unmonitored if its TLS private keys expired. It’s easy to see why: halting all network traffic when keys expire can be costly for business. Sure, it is inappropriate to have the security software in passthrough mode while there is an ongoing exfiltration attack, but that doesn’t change the fact that this was a feature it was designed with, not a random failure.

1. Boeing 737 MAX MCAS problems (2018–2019): “Flight control software errors contributed to two crashes, 346 deaths.”

The software did exactly what it was supposed to in response to the input it got: it assisted pilots with pointing the nose of the plane down. Did the pilots – or anyone else on the plane – want that in those situations? No. But the software did exactly what it was designed to do. It did not suddenly go from doing what it was designed to do to doing something which was not intended, so it did not fail. The angle of attack sensor failed (and there were plenty of design errors in both software and human protocols), but the software did not fail.

1. CrowdStrike outage (2024): “Faulty update crashed 8.5 million Windows systems globally, disrupted airlines, hospitals, banks.”

The update responsible reconfigured the Falcon® troja^H security software with a new emergent feature: if any named pipes were opened, the computer would reboot. Nobody intended for this feature to be in the software, because Windows opens named pipes in the process of booting, but it was in there none the less. It didn’t suddenly start happening; it was in that update when it left the factory.

1. Language model unreliability (2025): “The LLM was asked for ten high-profile software-contributed losses, and it produced a list where Knight Capital featured twice, leaving me without a tenth item so I’m putting the LLM itself in that spot.”

This is also not a random failure, even if at first it might seem that way. This problem is built into the architecture of the LLM, and always has been. The LLM is not designed to reliably follow prompts, however much their creators want to convince their prospective customers of that.[^7]

That’s ten software-contributed losses you’ve probably heard of, and in none of them software failed.

When people say the software failed, they usually don’t actually mean that the software failed, but rather what we have already seen:

• The software, as intentionally designed, contributed to a loss (this is the case when an explicitly requested feature rather directly causes the loss); or
• The software was designed with emergent features we did not want (this is the case when different software components interact to create a new feature).

We have seen the phrase emergent feature a few times now, so it makes sense to clear up what it means. An emergent feature is a feature designed into the system, but which does not have a specific component responsible for it. The emergent feature emerges from the interaction between components, or the interaction of the software and its environment. Emergent features are sometimes desired, sometimes unwanted, but either way, they are designed into the software just as much as explicit features. The reason we have to think about emergent features differently is that traditional analysis methods – which decompose the system into components – tend to miss them.[^8]

Traditional safety engineering focuses on failures

Traditional safety engineering, i.e. how most of us try to build robust things, focuses on failures. It answers questions such as:

• How long can this ball bearing run for before failing?
• In which ways can this valve fail? Can it be stuck closed?
• What happens to the reaction if the cooling circulation fails?

These are obviously related to safety, because if the ball bearing breaks, the pump will stop. If the valve gets stuck closed, circulation stops. If the cooling circulation fails, the pressure and temperature in the reaction chamber will go up. These are simple cause-and-effect relationships and we love them.

We’re also really good at dealing with them. We’re good at modeling component reliability[^9], building reliable components that do not fail[^10], and also at addressing problems that occur when components fail[^11].

Traditional component failure-based analysis might sound something like:

There are at least three problems with this analysis, though. First off, if one of the databases start to fill up with logs, it seems plausible that the other might also be exposed to the same erroneous commands and do the same thing. Thus their failures aren’t totally independent, and the 4 % figure is too low.

Second, if the database software contains an emergent feature where it fills up the disk space with logs, we don’t have to treat that as a random failure. We can handle that by figuring out under which conditions it is triggered and then working on avoiding those conditions rather than accepting it as a random failure event. We can do this because software never fails; it always does exactly what we told it to.

But the biggest problem with the analysis is that it doesn’t account for interaction problems between the two replicae. Are there conditions in which the replication process itself can make the database unavailable? Could both instances think they’re primaries and accept writes, only to later connect and discover the discrepancies, or will the system degrade into a read-only mode when the instances cannot connect to each other? The analysis completely ignores these interaction problems when it says the total failure rate of two 20 % databases is 4 %.

Traditional safety engineering focuses on the wrong thing

The problem we’re solving with safety engineering is avoiding losses. Traditional safety engineering avoids losses by looking at component failures. This seems sensible, because component failure can lead to loss, although components can also fail without loss – this is common when there is redundancy in place already. Thus we can say two things about analysis focused on failure:

• A component failure may contribute to a loss, but
• A component failure does not necessarily lead to a loss.

It’s easy to think that by focusing on failures we are, then, over-designing, i.e. covering more ground than we strictly have to. This would be a good thing when it comes to building robust systems! But that’s not really what happens. It turns out we are focusing on the wrong thing.

Traditional component failure-based analysis comfortably handles the case where components fail and the situation goes bad. It also handles the case where components fail and the situation is fine. It gets there by asking questions about failures first, and looking for which of them cause losses later.

What about that quadrant where no component has failed, yet things are terrible? The traditionalists say that can never happen, but we have just seen it happen in ten high-profile software accidents. The software didn’t fail. Software is design-stuff. It doesn’t fail ever. But it sure can contribute to a loss, even when it works perfectly as designed.

The replication configuration in the database analysis above, despite working exactly as intended, could interact unfavourably with our system[^12], and lead to losses even when the database has not failed. This isn’t at all reflected by that 4 % probability which only concerned failure events. This is a loss that happens in some of the other 96 % of the time when the database works as intended.

There is a whole category of issues that can contribute to expensive losses that are entirely ignored by traditional methods that focus on failure. These exist in any complex system, but they are particularly prevalent in software systems.

We need methods better than traditional failure-based analysis to solve these issues. This is where system-theoretic methods like STPA step in. If you agree, you might enjoy articles like the introduction to hazardous states and accidents, or maybe the STPA analysis of the AWS DynamoDB outage.

Think of wetware as software

As a final point, I want to mention that it’s useful to think of humans – wetware – the way we think of software. Sure, the hardware in a human can fail (lungs fill up with fluid, heart enters atrial fibrillation, brain haemorrhage, etc.) but the decisions humans make are functions of their training and environment. Importantly, human decisions are not random.

Some of you may have had trouble accepting that software does not behave randomly. A lot of you will have trouble accepting that humans don’t behave randomly. Remember the context in which you read this: building robust systems.

If we want to build robust systems in which humans play a part, we cannot write off human decisions as random. We must investigate what made the human act that way because that’s the only way we can improve system robustness. I’m not making a philosophical statement by saying human behaviour isn’t random – it’s a practical standpoint for the purpose of safety engineering. Human decision-making never fails. Instead, human error is a symptom of a system that needs to be redesigned.

Nobody stands before a choice where one action is labeled “correct” and the other “mistake” and choose to perform the mistake. All mistakes look equally or more correct to the person at the time, for whatever reason. If we build a system which does not tolerate human mistakes, then whose fault is it really when it blows up?

It might seem like a good idea to model human behaviour as random and treat it statistically (build in failsafes and automation to catch bad human behaviour) but human factors research has repeatedly shown that’s not the way to integrate humans in a system. It doesn’t work. Humans are brought into the system when it’s too complex to automate. We rely on the human ability to handle complexity, but then we need to give the human the best possible conditions in order to achieve robustness.

So instead of modelling humans statistically, we have to investigate how humans make decisions and adapt the system to improve the process of collaboration between man and machine. I could say a lot more about human error, but that’d be a separate article. In the mean time, you may enjoy what I learned from the recent Cloudflare outage.

Generated for the Product Growth Program.